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Recommendations for Data Review & Approval Guidelines 


On 9/22/98, the Data Review & Approval Guidelines Team was chartered. The deliverables 
were to define and document the scope of responsibilities and criteria for reviewing test results, 
rejecting test results, performing rechecks, and documenting data rejection. These functions 
were to be reviewed for the testing technicians, laboratory technical leaders, and receiving and 
reporting analysts (attachment 1). The team consisted of Cindy Boswell (leader), Robin 
Holleman, Saundra Hughes, Doris Johnson, Susan Laffoon, Barbara Ward, and Martha Payne, 
with Wade Mokarry as the sponsor (replacing Becky Tobey). The following is a review of our 
recommendations, presented on 11/4/98. 

Focus Statements 

The main function of the Product Testing Laboratory is to provide our collaborators with 
accurate and timely data that assist them in making decisions about our products. Therefore, we 
want to focus on the integrity of our data, so that others can ensure the integrity of our products. 
The statements on attac hme nt 2 served to keep us focused on the main function at each of the 
levels of data review and approval under consideration. 

Team Process 

The team met approximately 13 times between September 22 and November 3, 1998. We began 
by reviewing the current data review and approval processes for each lab area, by test or group of 
tests, and identified guidelines that were in place. We identified guidelines that were, in our 
opinion, subjective, inconsistent, or questionable; guidelines that could be improved and 
streamlined; and areas for which no guidelines existed. After the identified issues were 
categorized and addressed, recommendations were developed. Attachments 3 through 5 illustrate 
the accountabilities and recommendations for each of the three levels under consideration. 

Testing Technicians 

For the testing technicians to ensure the correct and consistent calibration and testing based on 
procedures, work instructions, checklists, and control charts, we recommend that work 
instructions be at all stations. For this to occur, we recommend that Natural Management Teams 
within each laboratory ensure that all test instructions are correct, usable, and posted at the test 
stations. Natural Management Teams may charter laboratory sub-teams to accomplish this. In 
addition, we recommend that control chart guidelines be developed by a sub-team so that testing 
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technicians will know how to detect control chart alarms and the procedures to follow when they 
are detected. Some guidelines may be laboratory-specific. 

For the testing technicians to help ensure that samples are not mixed up, we recommend that they 
make certain that the correct sample is tested. At present, there is no way for this to occur 
because only sample codes, which are difficult to decipher, appear on the barcodes. We 
recommend that IS be requested to modify barcodes and groupsheets so that brand names can 
appear for technicians to compare their samples to the paperwork. We also recommend that the 
testing technicians ensure all associated analyses are requested for the samples as per the request. 

We recommend that the testing technicians review test results for too few replicates and for 
inconsistent numbers, but NOT by comparing the sample standard deviation to the computerized 
standard deviation limits. To “review data for inconsistent numbers” is still a somewhat 
subjective statement which can be explained by several examples. If 14 of the 15 replicates of a 
physical property (Total RTD, for example) are between 35 and 40, and one result is a 90, this is 
an inconsistency in the data which may be a result of one cigarette being damaged or tested 
improperly. If approximately half of the data are between 35 and 40 and the other half are 
between 85 and 90, this may be due to two samples being mixed together. If the numbers 
consistently decrease drastically within a sample, for example from 85 to 60, this may be due to 
an equipment problem. An inconsistency in the data should be considered a flag to investigate 
for an assignable cause, not a criteria to indiscriminately reject data or recheck a sample. 

If an assignable cause is determined, either as a result of detecting inconsistent data, as a result of 
a control chart alarm, or any other sample problem, mix-up or equipment malfunction, we 
recommend that the testing technicians have the authority to reject data. We further recommend 
that the rejected data and its cause for rejection is documented in computer notes editing. If 
enough data are rejected to require a recheck, the testing technicians have the authority to request 
a recheck, or ask the technical leader to do so depending on the procedures set forth in the 
laboratory. To accomplish these tasks, we recommend that the Natural Management Teams 
within the laboratories charter sub-teams to begin developing lists of assignable causes for each 
test. The sub-teams should consist of testing technicians since they are the ones closest to the 
process and they know the equipment and the test better than most. The lists of assignable 
causes would be a living document, that would grow as necessary. 

In addition, a team should be assigned to develop guidelines for consistent use of the notes 
editing function, and to ensure that the notes editing function is available for all technicians to 
access during testing. This includes, but is not limited to, requesting a PC for the lucite room 
(where assignable causes can be detected on the TPM pads during transfer), and ensuring that the 
notes editing screen can be minimized on the same PC on which testing is performed in the 
Physical Properties laboratoiy. Additionally, this sub-team should investigate the computer trail 
for the notes editing fields to determine when notes should be captured as “lab-only”, where the 
notes reside in the computer, and whether modifications need to be made to this function. 

Certain other data handling issues can be resolved to reduce unnecessary work for the testing 
technicians. If a port’s TPM and Puff Count data are rejected (via the “bad port” routine) due to 
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equipment malfunction or sample problem, the computer system should automatically set the 
corresponding Nicotine and Water values to “missing.” Also, if a water result is negative due to 
detection being lower than the standard, this value should automatically be set to “0.0000.” 

Technical Leaders 

The PTL computer system currently contains an Automatic Data Approval Program (ADAP): 
that is, the computer’s first check of standard deviation, MAD outlier program (if necessary, 
based on first standard deviation check), and second standard deviation check (if gone through 
MAD). While it is necessary to have such a system so that not all data must be manually 
reviewed, some aspects of this program may be addressed in the near future to ensure that our 
data are being evaluated appropriately. Regardless of what changes may occur to this system, we 
recommend that the technical leaders only review data that have been subjected to but not 
approved by the ADAP. Currently, this would include any test that is listed on the Status 50 
report which lists tests not approved by ADAP, and any test on the BrioQuery® To Do report 
which shows tests not completed due to too few measurements. 

To effectively use these two reports, we recommend that all tests be performed with a minimum 
of three replicates so that all appropriate data can be subjected to ADAP. In addition, a small 
sub-team should be chartered to assess the current standard deviation limits in the ADAP system 
and develop a scientific algorithm for determining objective limits. The To Do report currently 
runs only on domestic samples. An individual should be assigned the project to adapt the 
program to run against all samples in all laboratories. 

We recommend that the above unapproved data results are reviewed by first investigating for 
assignable causes. An assignable cause may be strongly suspected although it may never be 
unequivocally proven. For example, in analyzing a smoking group sheet, it may be obvious from 
the data that two or more samples were smoked in switched positions during a second run, 
although it may never be proven. If an assignable cause is found or suspected with strong 
enough evidence, the samples may be rechecked. Samples may also be rechecked if there are not 
enough data values, as determined by the test definition. If the standard deviation of a sample is 
high enough to cause it not to be approved, and no assignable cause is found, the sample may be 
rechecked if the standard deviation is at least twice the acceptable limit. This will confirm that 
the variation is due to the sample and not the testing process. 

If a technical leader requests a recheck, all original data values are automatically rejected by the 
computer. The technical leader is to enter the reason for the recheck and assignable cause (if 
appropriate) into both notes editing and the laboratory’s recheck notebook. We recommend that 
all rechecked data be subjected to ADAP but NOT be automatically approved by the program 
(this may already be the procedure but we have found situations where it has not proceeded this 
way). This would ensure that all rechecked data are reviewed by the technical leader to resolve 
the original reason for the recheck. If the reason for the recheck was high standard deviation, and 
the rechecked data also have a high standard deviation, unless an assignable cause is found, the 
data are to be approved regardless of the resulting standard deviation (no second recheck for 
double the standard deviation limit). The original high standard deviation is therefore confirmed 
to be a result of sample variation, not testing variation. 
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Finally, we recommend that the technical leaders review the work instructions with the testing 
technicians AT LEAST twice a year, or more frequently, as deemed necessary by laboratory 
technical leadership. Following each review, each testing technician is to be observed 
performing the procedures and their proper use of the work instruction is to be documented. 
Likewise, improper use of the work instruction is to be corrected and documented. 

Receiving and Reporting 

The sample receiving function provides the first opportunity for PTL to ensure that our samples 
are entered into the system correctly, tests are requested correctly and the appropriate paperwork 
is generated. This is the first opportunity to avoid sample mix-ups in the testing process. It is 
also important that the analysts accurately obtain and record as much relevant information from 
the collaborator as possible. Especially in the case of developmental samples, information 
regarding the program or project may prove useful in the final review of the request. 

The receiving and reporting analysts monitor the request in process to ensure that testing is 
completed in a timely manner. Once testing is completed, they review the Preliminary Report 
and other data reports to ensure the integrity of the request prior to the release of data to the 
collaborator. We recommend that the following guidelines be utilized in reviewing the request; 

A. Review of Standard Deviation Warnings 

Warnings are printed on the Preliminary Report to denote test samples resulting in 
a standard deviation higher than the pre-determined acceptable limit. These 
situations should have already been reviewed at the technical leader level via the 
use of the Status 50 report. If so, they may have already been rechecked, either as 
a result of a documented assignable cause or to confirm a grossly high standard 
deviation (twice the acceptable limit). To assist the receiving and reporting 
analysts, we recommend that any recheck result printed on the Preliminary Report 
be flagged or identified in some way as a rechecked value. (Notes editing should 
also be referenced by the receiving and reporting analyst to view comments from 
the technical leader or testing technicians.) Since standard deviation warnings 
should have already been reviewed by a technical leader, we recommend that 
receiving and reporting analysts only review samples with standard deviations 
twice the acceptable limit. If a test sample still has a high standard deviation after 
being rechecked, this second high standard deviation confirms that the variation is 
due to the sample and not the process, and no further investigation is required. If 
a high standard deviation sample (twice the limit) has not yet been rechecked, we 
recommend that the receiving and reporting analysts take the following action: 

1- Available Information - Assess any information obtained regarding the 
program or project. Special projects may indicate developmental work which 
could explain greater variation than usually observed. If available information 
explains the variation, no further action is required. 

2. Investigate and/or Confirm - If there is no information to explain the sample 
variation, the receiving and reporting analyst should initiate investigation for 
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assignable cause. This should include looking for evidence of a sample mix- 
up, testing errors, or system errors. If an assignable cause is determined, it 
should be recorded in notes editing. Regardless of whether an assignable 
cause is determined, if sample is still available, a recheck should be initiated 
to either test the correct sample (in case of a mix-up), correctly test the sample 
(in case of a system or testing error) or confirm the sample variation (to verify 
that it is not due to testing variation). 

B. Review of Averages 

Samples received for testing in PTL may have targets set for production, historical 

data, both, or neither. We recommend the average test results be reviewed 

relative to this available information as follows: 

1 • Samples with Targets - If targets are available for certain test parameters, the 
receiving and reporting analysts should review the test averages against these 
targets according to the guidelines in attachment 6. This serves primarily as a 
check on the accuracy of the sample. 

2. Samples with Historical Data - If historical data are available for the samples, 
the receiving and reporting analysts should review the test averages against a 
3-standard deviation (historical) confidence interval around the historical 
average. This creates an approximation to a control chart for the sample and 
serves primarily as a check on the trend of the sample. 

3. Samples with Both Targets and Historical Data - It is important to maintain a 
distinction between the guidelines used for targets and those used for trends; 
the target guidelines are related to production tolerances (for smoking 
parameters) and these tolerances can not be applied to trends. However, the 
results of comparing the average to the target should be used in conjunction 
with the results of comparing the average to the trend; the two provide 
different information that needs to be assimilated prior to investigation. 

Example - If a tar average “fails” the target comparison test, the tar value may 
be within the historical 3-standard deviation confidence interval, but results 
have been steadily dropping. Therefore, the test average is justifiable and the 
deviation from target should be considered a product problem. However, if a 
test average “fails” the target comparison test, does not fall within the 3- 
standard deviation limit, and the data have not been trending toward this 
value, there may have been a sample or testing problem in PTL. 

4. Samples with Neither Targets nor Historical Data ( developmental, etc. - ) - If a 
sample parameter has no target and no historical data, the averages may be 
assessed based on information from the collaborator. Special programs or 
projects may include certain models with loose targets for testing parameters. 
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C. Confirmation of Averages 

If reviews of test averages result in outcomes that are questionable, we 

recommend that the receiving and reporting analysts take the following steps: 

1. Determine if relationships between test parameters confirm the test averages. 

2. Determine if available information from the collaborator regarding the 
program or project confirm the test results. 

3. Determine if the test results have already been rechecked which would serve 
as a confirmation of the results. 

4. Determine if an assignable cause can be found (sample mix-up, system or 
testing error). If found, and enough sample remains, a recheck should be 
initiated. 

5. If none of the above steps verify the average, and a recheck has not yet been 
performed but enough sample remains to do so, a recheck should be initiated 
to ensure that we have correctly tested the correct sample. Additionally, 
appropriate documentation needs to be recorded in notes editing. 

6. If appropriate, the collaborator should be contacted to discuss the discrepancy 
in the results. 


lecific Tasks 


Attachment 7 outlines the specific tasks that need to be accomplished in order to put all the 
recommendations in place. Each task has been prioritized and is assigned either an individual or 
group that this team recommends be chartered to complete it. The priorities were assigned based 
on a combination of how much effort we feel the task should require and how important the 
result is. The first six items are prioritized with an “*” because we feel that, if accepted, these 
items require so little effort on the part of PTL employees that they should be pursued 
irrespective of the other items. Each line-item also contains an estimate of how long we feel it 
should take to complete. 


If you have any further questions, please feel free to contact any one of us. 


Attachments 

Distribution: CC: 

Liz Chambers Charlene Callicutt 

Gloria Hicks-White Becky Tobey 

Wade Mokarry 
Ken Podraza 
Brenda Strang 
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